Papers with model scalability
MoFE: Mixture of Frozen Experts Architecture (2025.naacl-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are characterized by their immense size, often consisting of at least one billion parameters. |
| Approach: | They propose a mixture of Frozen Experts architecture that integrates PEFT and MoE to enhance both training efficiency and model scalability. |
| Outcome: | The proposed architecture outperforms other methods while achieving the highest efficiency. |
Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training (2025.naacl-long)
Copied to clipboard
| Challenge: | Current PP methods face severe bottlenecks, including pipeline bubbles and memory footprint. |
| Approach: | They propose a sequence-level one-forward-one-backward (1F1B) PP method for training LLMs on long sequences with high throughput and memory efficiency. |
| Outcome: | The proposed method achieves 1.14X training throughput with half memory footprint compared to baseline methods . it trains an LLM with 30B parameters on sequences up to 64k tokens using 64X NVIDIA A100 GPUs . |
A Dataset for Hyper-Relational Extraction and a Cube-Filling Approach (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods do not consider qualifier attributes for each relation triplet, such as time, quantity or location. |
| Approach: | They propose a hyper-relational extraction task to extract more specific facts from text using qualifiers. |
| Outcome: | The proposed model outperforms baselines and reveal possible directions for future research. |